Query entity-experience classification

ABSTRACT

The technology described herein makes improved use of limited screen space on a search results page by determining whether to present a question-and-answer experience and/or an entity details experience. This determination effects the amount of information presented and the format in which it is presented. In general, the question-and-answer experience provides less information and is more targeted to a question and query terms other than the entity. In contrast, the entity details experience provides more information about the entity that is not tailored to the query beyond the entity being included in the query. In one aspect, the determination of whether to show a question-and-answer experience and/or an entity details experience is based, at least in part, on an entity-details intent classification score (“intent classification score”) generated by a machine classification system. The classification score may be processed in combination with other criteria to make a final determination.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is continuation of U.S. patent application Ser. No.16/536,827, filed on Aug. 9, 2019, entitled “QUERY ENTITY-EXPERIENCECLASSIFICATION,” the entirety of which is hereby incorporated byreference.

BACKGROUND

Search engines receive queries and provide search results that areresponsive to a query provided by a user. The search engine can processthe query, user data, contextual data, and other inputs to identify themost relevant content for the particular query. The content can bepresented to the user in several different forms on a search resultspage. The content can be presented as links to webpages, aquestion-and-answer experience, an entity details experience, or in someother form. The search results page has a limited amount of screenspace, especially when the results are presented on a mobile phone orother device with a comparatively small screen. Users become frustratedwhen unwanted search results and or user experiences are presented,instead of, or even in addition to, the search results and userexperiences sought.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used in isolation as an aid in determining the scope of the claimedsubject matter.

Aspects of the technology described herein make improved use of limitedscreen space on a search results page by determining whether to presenta question-and-answer experience and/or an entity details experience.This determination effects the amount of information presented and theformat in which it is presented. In general, the question-and-answerexperience provides less information and is more targeted to a questionand/or query terms other than the entity. In contrast, the entitydetails experience provides more information about the entity that isnot tailored to the query beyond the entity being included in the query.The search results page may be a webpage displayed through a browser.The search results page may also be a page displayed through anapplication, such as a personal assistant application, navigationapplication, shopping application, and the like.

In one aspect, the determination of whether to show aquestion-and-answer experience and/or an entity details experience isbased, at least in part, on an entity-details intent classificationscore (“intent classification score”) generated by a machineclassification system, such as a neural network or support vectormachine (SVP). The machine classification system is trained usingqueries labeled as having an entity-details intent or not having anentity-details intent. Once trained, the machine classification systemcan assign an intent classification score to an unlabeled query. Theintent classification score can be a confidence score indicating adegree of confidence the machine classification system assigns to thequery that an entity-details experience should be shown in response tothe query. Conceptually, a high score means that the new query has ahigh level of similarity to training data labeled with an entity-detailsintent. A low score means the opposite.

The intent score may be used in combination with various rules todetermine whether an available question-and-answer experience ispresented to the user. In one aspect, two different intent-score-basedrules are used to determine whether an entity-details intent exists. Ifeither of the two intent-score-based rules determines thatentity-details intent exists, then an available Q&A experience is notpresented and only the entity-details experience is presented. Asdescribed subsequently, other rules that do not rely on the intent scoremay be used to determine that entity-details intent exists.

In addition to the intent-score-based rules, content-based rules may beused to determine that an entity-details intent exists. In one aspect,two different content-based rules are used to determine whether anentity-details intent exists. If either of the two content-based rulesdetermines that entity-details intent exists, then an available Q&Aexperience is not presented and only the entity-details experience ispresented.

Thus, aspects of the technology may use four or more different rules tofind an entity-details intent. If any one or more of the rules issatisfied, then the intent is found. The rules may be evaluated inseries or in parallel. If in series, then subsequent rules in the seriesneed not be evaluated when a precedent rule finds an entity-detailsintent. If in parallel, then processing of other rules may stop when arule finds an entity-details intent.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the invention are described in detail below with reference tothe attached drawing figures, wherein:

FIG. 1 is a block diagram of an example operating environment suitablefor implementing aspects of the technology;

FIG. 2 is a diagram showing a format selection environment, according toan aspect of the technology described herein;

FIG. 3 is a diagram showing a question and answer experience, accordingto an aspect of the technology described herein;

FIG. 4 is a diagram showing an entity detail experience, according to anaspect of the technology described herein;

FIGS. 5-7 depict flow diagrams of methods for providing relevant searchresult content in an experience format that is optimized to the query,in accordance with an aspect of the technology; and

FIG. 8 is a block diagram of an exemplary computing environment suitablefor use in implementing an aspect of the technology.

DETAILED DESCRIPTION

The subject matter of aspects of the technology is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Aspects of the technology described herein make improved use of limitedscreen space on a search results page by determining whether to presenta question-and-answer experience and/or an entity details experience.This determination affects the amount of information presented and theformat in which it is presented. In general, the question-and-answerexperience provides less information and is more targeted to a questionand/or query terms other than the entity. In contrast, the entitydetails experience provides more information about the entity that isnot tailored to the query beyond the entity being included in the query.The search results page may be a webpage displayed through a browser.The search results page may also be a page displayed through anapplication, such as a personal assistant application, navigationapplication, shopping application, and the like.

A question-and-answer experience attempts to provide a concise answer toa question posed in the query about an entity. The information in theanswer may be sourced from many different locations, such as knowledgebases, dictionaries, and webpages. The content provided about the entityin the answer is specific to the query, rather than general. The answercontent is selected for relevance to terms, other than the entity, inthe query.

An entity details experience is generated from a knowledge base andpresents multiple attribute values for the entity. The attributes arestandard for entities of a type, such as city, location, or person. Theattribute value is presented and associated with an attribute label,such as birthdate. For example, the attributes shown for two actors willlikely be similar, but, of course, with different attribute values.Attributes can differ for entities of the same type when attribute valuedata is not available in the knowledge base for one of the entities. Theattributes shown will be the same for any query that includes theentity.

In one aspect, the determination of whether to show aquestion-and-answer experience and/or an entity details experience isbased, at least in part, on an entity-details intent classificationscore (“intent classification score”) generated by a machineclassification system, such as a neural network or support vectormachine (SVP). The machine classification system is trained usingqueries labeled as having an entity-details intent or not having anentity-details intent. Once trained, the machine classification systemcan assign an intent classification score to an unlabeled query. Theintent classification score can be a confidence score indicating adegree of confidence the machine classification system assigns to thequery that an entity-details experience should be shown in response tothe query. Conceptually, a high score means that the new query has ahigh level of similarity to training data labeled with an entity-detailsintent. A low score means the opposite.

Generating training data for the machine classification system can be anexpensive undertaking. In one aspect, the training data is obtained byhaving a person review and label a query. The number of training dataqueries needed to sufficiently train the machine classification systemcan be reduced if the training data comprises an optimal number of edgecases. Edge cases are those in the border area between having anentity-details intent and not having the intent. In other words, theedge cases are more ambiguous than queries with a clear-cutentity-details intent.

The technology described herein efficiently generates training datacomprising edge cases by first training a mini-classifier on acomparatively small set of data. Because of the small set of data, themini-classifier will not have the desired accuracy of a productionclassifier. However, the mini classifier can process a large number ofqueries, optionally taken from a query log of actual queries, and assignan intent score. The queries assigned an intent score within an edgezone, such as between 0.45 and 0.55 (on a 0 to 1 scale), can then beused as input to the labeling process. A person would then assign alabel for each of these “edge” queries during the labeling process. Thetraining data can also include strong positives and strong negatives.The output from the mini-classifier can also be used to identify strongnegatives and strong positives. The end result is an efficient mixtureof presumptively strong positives, strong negatives, and edge casequeries that can be used to generate training data. Using this mixtureas a feed to the labeling process can reduce the number of queries thatneed to be labeled to train a precise and robust machine classifier.

The machine classifier can take the entire query as input, but can alsotake other data derived from the query as additional or alternativeinput. For example, the parts of speech of each word in the query can beprovided as input. In another aspect, the total number of words in aquery can be provided as input. Question terms in the query, such aswho, what, when, where, why, and how, can be specifically identified. Anentity in the query can be assigned a broad role, such as a person,place, thing, concept, and the like. When available, a specific role forthe entity in the query can be identified, such as actor, singer,politician, and the like. Other preprocessing could be performed on thequery and provided as input to the machine classifier. The preprocessingcan be used both during training and production of the classifier.

Switching from how the intent score is generated to how it is used, theintent score may be used in combination with various rules to determinewhether an available question-and-answer experience is presented to theuser. In one aspect, two different intent-score based rules are used todetermine whether an entity-details intent exists. If either of the twointent-score based rules determines that entity-details intent exists,then an available Q&A experience is not presented and only theentity-details experience is presented. As described subsequently, otherrules that do not rely on the intent score may be used to determine thatentity-details intent exists.

The first intent-score based rule takes the intent score from theclassifier, usage information about the intent experience that could beshown in response to the query, and information about the Q&A experiencethat could be shown in response to the query as input. The informationabout the Q&A experience is a type classification. Q&A experiences canbe classified as many different types. Two types of particular interestare health information and lists. A health type of Q&A experienceprovides information related to a medical condition. For example,symptoms of scarlet fever may be presented in a Q&A experience inresponse to a query “scarlet fever.” A list of top grossing movies maybe presented as a Q&A experience in response to a query “top 10 moviesof 2018.” Other types of Q & A experiences include technicalinformation, financial information, date and time information, andweather. An exemplary date and time query could be, “when is Labor Day?”

In one aspect, the first intent-score based rule requires threedifferent criteria to be met in order to find that an entity-detailsintent exists. First, the intent score from the classifier needs to beabove a high threshold score. This score is described as “high” inrelation to the lower threshold used in the second rule. Second, thesearch engine usage data must show that the entity experience has beenpresented previously in response to a similar query. Third, the Q&Aexperience type must not be on a white list. The white list includes Q&Aexperience types that should always be shown, which means that anentity-details intent should not be found. The white list can include atleast the health type, time/date, and list type of experiences. If allthree criteria are satisfied, then an entity-details intent is found.

The second intent-score-based rule uses the same input as the first,however, instead of the Q&A experience type, the rule uses the Q&Aexperience source. In one aspect, the second intent-score based rulerequires three different criteria to be met in order to find that anentity-details intent exists. First, the intent score from theclassifier needs to be above a medium threshold score. This score isdescribed as “medium” in relation to the higher threshold used in thefirst rule. The medium threshold may be set to be above a range thatdefines an ambiguous intent. For example, 0.6 (on a 0 to 1 scale) couldbe the medium threshold. In other words, the classifier shows anentity-details intent, but the signal does not need to be strong.Second, the search engine usage data must show that the entityexperience has been presented previously in response to a similar query.Third, the source of the Q&A experience must not be classified as acurated reference source, such as Wikipedia or IMDB. The technologydescribed herein can use a list of curated reference sources and use alookup function to determine if this third aspect of the rule issatisfied. In another aspect, if the entity-experience and the Q&Aexperience are from the same source, then the source of the Q&Aexperience is classified as a curated reference source. If all threecriteria are satisfied, then an entity-details intent is found.

In addition to the intent-score-based rules described previously,content-based rules may be used to determine that an entity-detailsintent exists. In one aspect, two different content-based rules are usedto determine whether an entity-details intent exists. If either of thetwo content-based rules determines that entity-details intent exists,then an available Q&A experience is not presented and only theentity-details experience is presented. Optionally, the intent scorecould be used with either content-based rule in the form of alow-threshold. The content-based rules form a strong presumption thatthe entity-details intent is present, thus, a lower threshold (than thepreviously described high and medium thresholds) could be used.

The first content-based rule determines whether a title of the entityexperience matches the query. If the title matches the query, then anentity-details intent is found. The match may be exact, n-gram based,and/or account for misspellings using a minimal edit distance.

The second content-based rule generates a similarity score thatquantifies the similarity between the content of entity experience andthe content of Q&A experience. If the similarity score is above athreshold, then an entity-details intent is found. The entity experienceoften includes more features, thus, the similarity score could be basedon an amount of content within the Q&A experience that is also in theentity experience. In other words, the similar score may be generated ina way that does not penalize the entity experience for having additionalcontent.

Thus, aspects of the technology may use four different rules to find anentity-details intent. If any one or more of the rules is satisfied,then the intent is found. The rules may be evaluated in series or inparallel. If in series, then subsequent rules in the series need not beevaluated when a precedent rule finds an entity-details intent. If inparallel, then processing of other rules may stop when a rule finds anentity-details intent.

In addition to the rules mentioned, the technology can use a hotfix. Thehotfix creates a list of queries that always result in a particularexperience, such as showing the Q&A experience. The hotfix servers is awork-around for popular queries that, for whatever reason, produce anundesirable result. The hotfix list could be evaluated as a first stepas a gatekeeper to the four previously described rules.

Having briefly described an overview of aspects of the technologydescribed herein, an exemplary operating environment in which aspects ofthe technology described herein may be implemented is described below.

Turning now to FIG. 1 , a block diagram is provided showing an operatingenvironment 100 in which aspects of the present disclosure may beemployed. It should be understood that this and other arrangementsdescribed herein are set forth only as examples. Other arrangements andelements (e.g., machines, interfaces, functions, orders, and groupingsof functions) can be used in addition to or instead of those shown, andsome elements may be omitted altogether for the sake of clarity.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distributed components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed by oneor more entities may be carried out by hardware, firmware, and/orsoftware. For instance, some functions may be carried out by a processorexecuting instructions stored in memory.

Among other components not shown, example operating environment 100includes a number of user devices, such as user devices 102 a and 102 bthrough 102 n; a number of data sources, such as data sources 104 a and104 b through 104 n; search server 106; and network 110. It should beunderstood that environment 100 shown in FIG. 1 is an example of onesuitable operating environment. Each of the components shown in FIG. 1may be implemented via any type of computing device, such as computingdevice 800, described in connection to FIG. 8 , for example. Thesecomponents may communicate with each other via network 110, which mayinclude, without limitation, one or more local area networks (LANs)and/or wide area networks (WANs). In exemplary implementations, network110 comprises the Internet and/or a cellular network, amongst any of avariety of possible public and/or private networks.

It should be understood that any number of user devices, servers, anddata sources may be employed within operating environment 100 within thescope of the present disclosure. Each may comprise a single device ormultiple devices cooperating in a distributed environment. For instance,search server 106 may be provided via multiple devices arranged in adistributed environment that collectively provide the functionalitydescribed herein. Additionally, other components not shown may also beincluded within the distributed environment.

User devices 102 a and 102 b through 102 n can be client devices on theclient-side of operating environment 100, while search server 106 can beon the server-side of operating environment 100. Server 106 can compriseserver-side software designed to work in conjunction with client-sidesoftware on user devices 102 a and 102 b through 102 n so as toimplement any combination of the features and functionalities discussedin the present disclosure. This division of operating environment 100 isprovided to illustrate one example of a suitable environment, and thereis no requirement for each implementation that any combination of searchserver 106 and user devices 102 a and 102 b through 102 n remain asseparate entities.

User devices 102 a and 102 b through 102 n may comprise any type ofcomputing device capable of use by a user. For example, in one aspect,user devices 102 a through 102 n may be the type of computing devicedescribed in relation to FIG. 8 herein. By way of example and notlimitation, a user device may be embodied as a personal computer (PC), alaptop computer, a mobile or mobile device, a smartphone, a tabletcomputer, a smart watch, a wearable computer, a personal digitalassistant (PDA), an MP3 player, global positioning system (GPS) ordevice, video player, handheld communications device, gaming device orsystem, entertainment system, vehicle computer system, embedded systemcontroller, remote control, appliance, consumer electronic device, aworkstation, or any combination of these delineated devices, or anyother suitable device where notifications can be presented. A user 105may be associated with one or more user devices. The user 105 maycommunicate with search server 106, data source 104 a and 104 b through104 n, through the user devices.

Data sources 104 a and 104 b through 104 n may comprise data sourcesand/or data systems, which are configured to make data available to anyof the various constituents of operating environment 100, or system 200described in connection to FIG. 2 . (For example, in one aspect, one ormore data sources 104 a through 104 n provide (or make available foraccessing) data for generating an entity-details experience or Q&Aexperience.) Data sources 104 a and 104 b through 104 n may be discretesearch server 106 or may be incorporated and/or integrated into at leastone of those components. The data sources 104 a through 104 n caninclude web servers that host web pages that can include contentresponsive to queries. The data sources 104 a through 104 n can includeone or more knowledge bases.

Operating environment 100 can be utilized to implement one or more ofthe components of system 200, described in FIG. 2 , including componentsfor receiving search queries, determining a query intent, and providingrelevant content in a Q&A experience and/or entity-details experience.

Referring now to FIG. 2 , with FIG. 1 , a block diagram is providedshowing aspects of an example computing system architecture suitable forimplementing an aspect of the technology and designated generally assystem 200. System 200 represents only one example of a suitablecomputing system architecture. Other arrangements and elements can beused in addition to or instead of those shown, and some elements may beomitted altogether for the sake of clarity. Further, as with operatingenvironment 100, many of the elements described herein are functionalentities that may be implemented as discrete or distributed componentsor in conjunction with other components, and in any suitable combinationand location.

At a high level, system 200 comprises a search service 210 that receivesa search query 201 and returns a search result page 202 that includes auser experience optimized to present relevant content. These userexperiences include an entity details experience and a Q&A experience.The entity-details intent determiner 240 can determine whether anentity-details intent exists for the query and provide outputinstructions based on the determination. The search service 210 may beembodied on one or more servers, such as search server 106.

Example system 200 includes the search service 210 (including itscomponents 212, 220, 230, and 240) and classifier trainer 250 (includingits components 252, 254, and 256). The search service 210 (and itscomponents) and classifier trainer 250 (and its components) may beembodied as a set of compiled computer instructions or functions,program modules, computer software services, or an arrangement ofprocesses carried out on one or more computer systems, such as computingdevice 800 described in connection to FIG. 8 , for example.

In one aspect, the functions performed by components of system 200 areassociated with one or more personal assistant applications, services,or routines. In particular, such applications, services, or routines mayoperate on one or more user devices (such as user device 102 a), servers(such as search server 106), may be distributed across one or more userdevices and servers, or be implemented in the cloud. Moreover, in someaspects, these components of system 200 may be distributed across anetwork, including one or more servers (such as server 106) and clientdevices (such as user device 102 a), in the cloud, or may reside on auser device such as user device 102 a. Moreover, these components,functions performed by these components, or services carried out bythese components may be implemented at appropriate abstraction layer(s)such as the operating system layer, application layer, hardware layer,etc., of the computing system(s). Alternatively, or in addition, thefunctionality of these components and/or the aspects of the technologydescribed herein can be performed, at least in part, by one or morehardware logic components. For example, and without limitation,illustrative types of hardware logic components that can be used includeField-programmable Gate Arrays (FPGAs), Application-specific IntegratedCircuits (ASICs), Application-specific Standard Products (ASSPs),System-on-a-chip systems (SOCs), Complex Programmable Logic Devices(CPLDs), etc. Additionally, although functionality is described hereinwith regards to specific components shown in example system 200, it iscontemplated that in some aspects, functionality of these components canbe shared or distributed across other components.

The search service 210 includes a query interface 212, a queryprocessing component 220, a search engine 230, and an entity-detailsintent determiner 240. The query processing component 220 includes agrammar identification component 222, a broad role identificationcomponent 224, a specific role identification component 226, a wordcount component 228, and a question identification component 229. Thesearch engine 230 includes a search log 232, a search index 234, aquestion-and-answer experience component 236, and an entity detailsexperience component 238. The entity-details intent determiner 240includes a classifier 242, a rule engine 244, an intent interface 246,and a hotfix component 248.

The search service 210 receives a search query 201 and returns a searchresults page 202 that includes a user experience optimized to presentrelevant content. The search results page may be a webpage displayedthrough a browser. The search results page may also be a page displayedthrough an application, such as a personal assistant application,navigation application, shopping application, and the like.

The query interface 212 receives a query and communicates it to othercomponents, such as the query processing component 220, the searchengine 230, and the entity-details intent determiner 240. The queryinterface 212 may generate a graphical user interface through which thequery is input, such as a search box on a web page. The query interface212 may also comprise an Application Program Interface (API) that letsapplications submit the query (and optionally other information, such asuser information, contextual information, and the like) to the searchservice 210. The query interface can also output search results, such asthe search results page 202.

The query processing component 220 identifies features of the query thatcan be provided to the classifier 242. The classifier 242 uses thefeatures as an input to generate an entity-details intent score.

The query processing component 220 includes a grammar identificationcomponent 222 that identifies the part of speech for each term in thequery. The output can be a sequence of speech parts or the terms in thequery mapped to the speech parts. Other output forms are possible.

The broad role identification component 224 may identify the entity andidentify a broad role for each term that identifies an entity. Forexample, the entity Tiger Woods could be identified as a person. In oneaspect, the broad role identification component 224 outputs a queryoverlay with the entity terms replaced by the broad role. Thus, “howmany golf tournaments has Tiger Woods won” becomes “how many golftournaments has [person] [person] won?” Other forms of output arepossible.

The specific role identification component 226 is similar to the broadrole identification, but provides more granular information whenpossible. For example, Tiger Woods could be a professional athlete or,more specifically, “a PGA golfer.”

A word count component 228 counts the words in the query. In one aspect,the word count component 228 counts spaces in the query to determine thewords. This means that the words do not need to be actual words, butcould be a group of characters.

A question identification component 229 identifies terms that indicate aquestion, such as who, what, when, where, why, which, how, and the like.This list is not comprehensive. A semantic analysis may be performed todetermine whether words, like “which,” indicate a question or not.

The search engine 230 identifies content that is responsive to the queryand typically provides the most relevant results. The results caninclude Q&A experiences and entity-detail experiences.

A search log 232 stores the queries entered by users, results returned,and click-through for the URIs included in the results. The queriesstored in the search logs 232 may include entity queries. In someembodiments, the search logs 232 stores a timestamp for each query. Thetimestamp represents the day, hour, minute, second, etc., that the queryis received. The search logs 232 store the number of queries received bythe search engine; number of clicks, hovers, etc., received from aclient device for each URI returned in response to the query; and atleast one identifier for each of the URIs interacted with by the user ofa client device. The search log 232 can identify Q&A experiences andentity-detail experiences shown in response to a query. The userinteraction with these experiences can also be recorded.

The search index 234 includes information about documents (e.g., webpages) that may be returned in a search result. The index 234 can takethe form of an inverted index, but other forms are possible. The index234 allows the search engine to identify documents that are responsiveto the query. The index 234 can include keywords and other informationthat describes a document for the purpose of determining its relevance.

The question-and-answer experience component 236 generates aquestion-and-answer experience, such as experience 300 shown in FIG. 3 .

The entity details experience component 238, generates entity-detailexperiences, such as experience 400 shown in FIG. 4 .

The entity-details intent determiner 240 determines whether an entityexperience, Q&A experience, or possibly both is the most relevantresponse to the query. The entity-details intent determiner 240 providesinstructions to the query interface 212, or some other component, topresent the optimal experience.

The classifier 242 generates an entity-details intent score. The scoreis a measure of how closely the query matches a criteria for showing theentity-details experience. In one aspect, the entity-details experienceshould be shown when two criteria are met. First, the query mentions atleast one entity literally. Second, the query is looking for thedefinition, description, or general information of an entity, which isliterarily mentioned in the query. An entity can be a real-world object,including person, location, organization, local, store/services/chain,movie, book, TV show, song, album, car model, product, etc., or aconcept including free will, attribution theory, scientific terms,medical, terminology, etc. For example, this query is an entity intentquery: {Incredibles 2}. And the following query is not, “Yuval NoahHarari book 2018.” The query is asking about the book, when correctlyunderstood, but the book entity “21 lessons for the 21st century” is notmentioned in the query, so it is not an entity-details intent query.

A classifier is used to determine how closely a query matches thiscriteria. The classifier includes a machine classification system, suchas a neural network or support vector machine (SVP). The machineclassification system is trained using queries labeled as having anentity-details intent or not having an entity-details intent. Oncetrained, the machine classification system can assign an intentclassification score to an unlabeled query. The intent classificationscore can be a confidence score indicating a degree of confidence themachine classification system assigns to the query that anentity-details experience should be shown in response to the query.Conceptually, a high score means that the new query has a high level ofsimilarity to training data labeled with an entity-details intent. A lowscore means the opposite.

The classifier 244 can take the query as input along with the queryprocessing 220 outputs, such as role identifications and word counts.

The rule engine 244 evaluates various rules to determine whether anentity-details intent exists. In one aspect, two differentintent-score-based rules are used to determine whether an entity-detailsintent exists. If either of the two intent-score-based rules determinesthat entity-details intent exists, then an available Q&A experience isnot presented and only the entity-details experience is presented. Asdescribed subsequently, other rules that do not rely on the intent scoremay be used to determine that entity-details intent exists.

The first intent-score based rule takes the intent score from theclassifier, usage information about the intent experience that could beshown in response to the query, and information about the Q&A experiencethat could be shown in response to the query as input. The informationabout the Q&A experience is a type classification. Q&A experiences canbe classified as many different types. Two types of particular interestare health information and lists. A health type of Q&A experienceprovides information related to a medical condition. For example,symptoms of scarlet fever may be presented in a Q&A experience inresponse to a query “scarlet fever.” A list of top grossing movies maybe presented as a Q&A experience in response to a query “top 10 moviesof 2018.” Other types of Q & A experiences include technicalinformation, financial information, date and time information, andweather. An exemplary date and time query could be, “when is Labor Day?”

In one aspect, the first intent-score based rule requires threedifferent criteria to be met in order to find that an entity-detailsintent exists. First, the intent score from the classifier needs to beabove a high threshold score. This score is described as “high” inrelation to the lower threshold used in the second rule. Second, thesearch engine usage data must show that the entity experience has beenpresented previously in response to a similar query. Third, the Q&Aexperience type must not be on a white list. The white list includes Q&Aexperience types that should always be shown, which means that anentity-details intent should not be found. The white list can include atleast the health type, time/date, and list type of experiences. If allthree criteria are satisfied, then an entity-details intent is found.

The second intent-score-based-rule uses the same input as the first,however, instead of the Q&A experience type, the rule uses the Q&Aexperience source. In one aspect, the second intent-score based rulerequires three different criteria to be met in order to find that anentity-details intent exists. First, the intent score from theclassifier needs to be above a medium threshold score. This score isdescribed as “medium” in relation to the higher threshold used in thefirst rule. The medium threshold may be set to be above a range thatdefines an ambiguous intent. For example, 0.6 (on a 0 to 1 scale) couldbe the medium threshold. In other words, the classifier shows anentity-details intent, but the signal does not need to be strong.Second, the search engine usage data must show that the entityexperience has been presented previously in response to a similar query.Third, the source of the Q&A experience must not be classified as acurated reference source, such as Wikipedia or IMDB. The technologydescribed herein can use a list of curated reference sources and use alookup function to determine if this third aspect of the rule issatisfied. In another aspect, if the entity-experience and the Q&Aexperience are from the same source, then the source of the Q&Aexperience is classified as a curated reference source. If all threecriteria are satisfied, then an entity-details intent is found.

In addition to the intent-score-based rules described previously,content-based rules may be used to determine that an entity-detailsintent exists. In one aspect, two different content-based rules are usedto determine whether an entity-details intent exists. If either of thetwo content-based rules determines that entity-details intent exists,then an available Q&A experience is not presented and only theentity-details experience is presented. Optionally, the intent scorecould be used with either content-based rule in the form of alow-threshold. The content-based rules form a strong presumption thatthe entity-details intent is present, thus, a lower threshold (than thepreviously described high and medium thresholds) could be used.

The first content-based rule determines whether a title of the entityexperience matches the query. If the title matches the query, then anentity-details intent is found. The match may be exact, n-gram based,and/or account for misspellings using a minimal edit distance.

The second content-based rule generates a similarity score thatquantifies the similarity between the content of entity experience andthe content of Q&A experience. If the similarity score is above athreshold, then an entity-details intent is found. The entity experienceoften includes more features, thus, the similarity score could be basedon an amount of content within the Q&A experience that is also in theentity experience. In other words, the similarity score may be generatedin a way that does not penalize the entity experience for havingadditional content.

Thus, aspects of the technology may use four different rules to find anentity-details intent. If any one or more of the rules is satisfied,then the intent is found. The rules may be evaluated in series or inparallel. If in series, then subsequent rules in the series need not beevaluated when a precedent rule finds an entity-details intent. If inparallel, then processing of other rules may stop when a rule finds anentity-details intent.

The intent interface 246 communicates the determined intent to consumersof this determination, such as the search engine 230 and the queryinterface 212. The search engine 230 may use this information toinstruct the Q&A experience component 236 and the entity detailsexperience component 238.

The hotfix component 248 acts as an override to other system components.Specific queries may be added to a hotfix list and liked to a desiredoutput, such as showing a Q&A experience, showing an entity detailsexperience, or showing both. When a received query matches a query onthe hotfix list, then the linked output is executed without furtheranalysis or regardless of the determination made by the rule engine 244.

The classifier trainer 250 trains the classifier 244. Generatingtraining data for the machine classification system can be an expensiveundertaking. In one aspect, the training data is obtained by having aperson review and label a query. The number of training data queriesneeded to sufficiently train the machine classification system can bereduced if the training data comprises an optimal number of edge cases.Edge cases are those in the border area between having an entity-detailsintent and not having the intent. In other words, the edge cases aremore ambiguous than queries with a clear-cut entity-details intent.

The technology described herein efficiently generates training datacomprising edge cases by first training a mini-classifier 254 on acomparatively small subset of the raw training data sets 252. Because ofthe small set of data, the mini-classifier 254 will not have the desiredaccuracy of a production classifier 244. However, the mini-classifier254 can process a large number of queries, optionally taken from a querylog of actual queries, and assign an intent score. The queries assignedan intent score within an edge zone, such as between 0.45 and 0.55 (on a0 to 1 scale), can then be used as input to the labeling process. Aperson would then assign a label for each of these “edge” queries duringthe labeling process. The training data can also include strongpositives and strong negatives. The output from the mini-classifier 254can also be used to identify strong negatives and strong positives. Theend result is an efficient mixture of presumptively strong positives,strong negatives, and edge case queries that can be used to generateoptimal training data 256. Using this mixture as a feed to the labelingprocess can reduce the number of queries that need to be labeled totrain a precise and robust machine classifier.

Different machine learning technologies are trained differently. Thefollowing example is for a neural network, but aspects of the technologyare not limited to use with a neural network. As used herein, a neuralnetwork comprises at least three operational layers. The three layerscan include an input layer, a hidden layer, and an output layer. Eachlayer comprises neurons. The input layer neurons receive the labeledquery (and associated preprocessing data) and pass data derived from thequery to neurons in the hidden layer. Neurons in the hidden layer passdata to neurons in the output layer. The output layer then produces anentity-details intent score. Different types of layers and networksconnect neurons in different ways.

Neurons have weights, an activation function that defines the output ofthe neuron given an input (including the weights), and an output. Theweights are the adjustable parameters that cause a network to produce acorrect output. For example, if the training query is labeled noentity-detail intent, then the correct output is to classify the queryas having not entity-details intent. The weights are adjusted duringtraining. Once trained, the weight associated with a given neuron canremain fixed. The other data passing between neurons can change inresponse to a given input (e.g., image). Retraining the network withadditional training queries can update one or more weights in one ormore neurons.

The neural network may include many more than three layers. Neuralnetworks with more than one hidden layer may be called deep neuralnetworks. Example neural networks that may be used with aspects of thetechnology described herein include, but are not limited to, multilayerperceptron (MLP) networks, convolutional neural networks (CNN),recursive neural networks, recurrent neural networks, and longshort-term memory (LSTM) (which is a type of recursive neural network).

In each type of deep model, training is used to fit the model output tothe training data. In particular, weights associated with each neuron inthe model can be updated through training. Originally, the model cancomprise random weight values that are adjusted during training. In oneaspect, the model is trained using backpropagation. The backpropagationprocess comprises a forward pass, a loss function, a backward pass, anda weight update. This process is repeated for each training query. Thegoal is to update the weights of each neuron (or other model component)to cause the model to produce an output that maps to the correct label.Each labeled query is input to the model and used to train it. Once asufficient number of training queries (and associated pre-processingdata) are fed to the model, then the training can stop. The classifier244 can then be used to generate text strings from unlabeled images ofrendered domain names.

The entity database 260 stores information on entities. The database maystore attributes about the entity. The attribute may indicate whetherthe entity is person, place, document, movie, song, etc. Additionalattributes may include a brief description of the entity. The entitydatabase 260 may be provided by a third party. Entities may beidentified from news stories or social media blogs. In one embodiment,the entity database 260 may be provided by a social media provider or acontact aggregator. In other embodiments, the entity database 260 alsostores the entities and the URIs that are mapped to the entities. TheURIs in the entity database 260 are extracted from search results thatare interacted with in response to a query specifying a correspondingentity. The interactions may include clicks, hovers, gestures, voicecommands, etc., received from a client device employed by a user. Thequery is processed by the search engine to return the search results.

FIGS. 3 and 4 illustrate a different information provided in aquestion-and-answer experience interface and an entity-detailsexperience respectively. Both are generated in response to the samequery, “Tom Cruise's wife.” As can be seen, each interface has pros andcons and may be optimized for different kinds of queries.

Turning now to FIG. 3 , a question-and-answer experience interface 300is shown. The interface 300 includes a search box 310 in which thequery, “Tom Cruise's wife” is written. The American actor Tom Cruise isthe only entity listed in this query and wife is the only other term inthe query. The interface heading 320 states Tom Cruise—Wife. Thequestion-and-answer experience attempts to answer the question, “who isTom Cruise's wife?” As can be seen, the only information provided aboutentity Tom Cruise pertains to the three women he has been married topreviously. The first answer section 322 shows a picture of Katie Holmeswith an indication she was married to Tom Cruise between 2006 and 2012.The second answer section 324 shows a picture of Mimi Rogers andprovides an indication she was married to Tom Cruise from 1987 to 1990.The third answer section 326 shows a picture of Nicole Kidman andindicates she was married to Tom Cruise from 1990 to 2001. The user whois only curious about Tom Cruise's wives would likely find thequestion-and-answer experience 300 optimal. The answer is providedwithout additional information.

Turning now to FIG. 4 , an entity-detail experience interface 400 isshown. The interface 400 provides a broad range of information about TomCruise. The interface 400 is also generated in response to the query“Tom Cruise—Wife.” The interface 400 includes an image section 410 withseveral images of Tom Cruise. The interface 400 includes an experiencetitle 422, a detailed role 424, a blurb 426, a list of spouses 428,birth information 430, height 432, network 434, partner 436, and a listof upcoming movies 438. Each of these sections includes a label andcorresponding attribute information. Taking the birth information 430 asan example, the label “born” is coupled with Jul. 3, 1962 (age 57). Ascan be seen, the entity-detail experience interface 400 includesinformation about Tom Cruise's wives along with other generalinformation that is not necessarily responsive to the query. Some usersmay find the question-and-answer experience 300 more responsive to aquery asking about Tom Cruise's wives. However, entity-detailsexperience 400 may be more responsive to a broad query such as simply,“Tom Cruise.”

FIG. 5 is a flow chart showing a method 500 for providing relevantsearch result content in an experience format that is optimized to aquery, in accordance with an aspect of the technology described herein.

At step 510, a query from a user is received from a user at a searchengine. The query includes an entity. For example, the query could be,“what are the names of Hamilton cast members?” In this example, themusical Hamilton is the entity. In one aspect, an entity is onlyconsidered to be “in the query” if it is explicitly recited, as withHamilton above. In contrast, “names” refers to cast members, who areentities, but the cast members are not considered entities for thepurpose of this technology because they are not listed by theirindividual names. Broadly, an entity can be a noun. An entity can be aperson, place, thing, philosophy, feeling, health condition, mentalstate, or the like. Practically, the technology may only identifyentities listed in a knowledge base or some other source. Thus, a queryincluding the name of a person who is not well-known may not beidentified as an entity.

At step 520, an entity-details intent classification score is determinedfor the query using a machine classifier. The machine classificationsystem can be a neural network, support vector machine (SVP), or usesome other technology. Essentially, the score is a measure of howclosely the query matches training queries labeled as having anentity-details intent. The machine classification system is trainedusing queries labeled as having an entity-details intent or not havingan entity-details intent. Once trained, the machine classificationsystem can assign an intent classification score to an unlabeled query.The intent classification score can be a confidence score indicating adegree of confidence the machine classification system assigns to thequery that an entity-details experience should be shown in response tothe query. Conceptually, a high score means that the new query has ahigh level of similarity to training data labeled with an entity-detailsintent. A low score means the opposite.

At step 530, information about a question-and-answer experience thatwould be presented by the search engine in response to the query isidentified. The information can include a source from which thequestion-and-answer experience is generated, a classification for thesource, and other information. The information can also include acategory or type of question-and-answer experience generated. The typecan be form-based, for example, lists, or content category-based, suchas healthcare related.

At step 540, an entity-details intent is determined to exist for thequery because the entity-details intent classification score is above athreshold score and the information about the question-and-answerexperience satisfies a criteria. The criteria may be that the Q&Aexperience type must not be on a white list. The white list includes Q&Aexperience types that should always be shown, which means that anentity-details intent should not be found. The white list can include atleast the health type, date/time type, and list type of experiences. Ifall three criteria are satisfied, then an entity-details intent isfound. An additional criteria might be that the search log shows that anentity-details experience has been presented previously in response tothe same query or a similar query.

In another aspects, the criteria is that the source of the Q&Aexperience must not be classified as a curated reference source, such asWikipedia or IMDB. The technology described herein can use a list ofcurated reference sources and use a lookup function to determine if thisthird aspect of the rule is satisfied. In another aspect, if theentity-experience and the Q&A experience are from the same source, thenthe source of the Q&A experience is classified as a curated referencesource. An additional criteria might be that the search log shows thatan entity-details experience has been presented previously in responseto the same query or a similar query.

The two criteria described previously could be combined with differentthreshold scores. In one aspect, the first criteria uses a higherthreshold score than the second.

At step 550, an entity-details experience is output to the user. Theentity-details experience identifies the entity and multiple generalattributes that are not selected based on content in the query. Anexemplary entity-details experience has been described previously withreference to FIG. 4 .

FIG. 6 is a flow chart showing a method 600 for providing relevantsearch result content in an experience format that is optimized to aquery, in accordance with an aspect of the technology described herein.

At step 610, a query from a user is received from a user at a searchengine. The query includes an entity. For example, the query could be,“what are the names of Hamilton cast members?” In this example, themusical Hamilton is the entity. In one aspect, an entity is onlyconsidered to be “in the query” if it is explicitly recited, as withHamilton above. In contrast, “names” refers to cast members, who areentities, but the cast members are not considered entities for thepurpose of this technology because they are not listed by theirindividual names. Broadly, an entity can be a noun. An entity can be aperson, place, thing, philosophy, feeling, health condition, mentalstate, or the like. Practically, the technology may only identifyentities listed in a knowledge base or some other source. Thus, a queryincluding the name of a person who is not well-known may not beidentified as an entity.

At step 620, the entity within the query is identified. An entity canrefer to a type of person such as an author, politician, or sportsplayer; a type of product such as a movie, book, or a consumer good; ora type of place such as a restaurant, hotel, recreation area, or retailstore. Entity identification can use machine learning. Different machinelearning technologies may be used, such as with conditional randomfields. In one aspect, domain-based identifiers are used. Eachdomain-based identifier is trained to recognize different types ofentities. One identifier could be trained to recognize people, anotherbusinesses, another books, another movies, etc.

At step 630, information about an entity-details experience that wouldbe presented by the search engine in response to the query isidentified. The information can include the title of the experience,content in the experience, and the like.

At step 640, an entity-details intent is determined to exist for thequery because the information about the entity-details experiencesatisfies a criteria. The first criteria can be whether a title of theentity experience matches the query. If the title matches the query,then an entity-details intent is found. The match may be exact, n-grambased, and/or account for misspellings using a minimal edit distance.

The second criteria determines whether a similarity score thatquantifies the similarity between the content of entity experience andthe content of Q&A experience is within a threshold closeness. If thesimilarity score is above a threshold, then an entity-details intent isfound. The entity experience often includes more features, thus, thesimilarity score could be based on an amount of content within the Q&Aexperience that is also in the entity experience. In other words, thesimilarity score may be generated in a way that does not penalize theentity experience for having additional content.

At step 650, an entity-details experience to the user is output. Theentity-details experience identifies the entity and multiple generalattributes that are not selected based on content in the query. Anexemplary entity-details experience has been described previously withreference to FIG. 4 .

FIG. 7 is a flow chart showing a method 700 for providing relevantsearch result content in an experience format that is optimized to aquery, in accordance with an aspect of the technology described herein.

At step 710, a query from a user is received at a search engine. Thequery includes an entity. For example, the query could be, “what are thenames of Hamilton cast members?” In this example, the musical Hamiltonis the entity. In one aspect, an entity is only considered to be “in thequery” if it is explicitly recited, as with Hamilton above. In contrast,“names” refers to cast members, who are entities, but the cast membersare not considered entities for the purpose of this technology becausethey are not listed by their individual names. Broadly, an entity can bea noun. An entity can be a person, place, thing, philosophy, feeling,health condition, mental state, or the like. Practically, the technologymay only identify entities listed in a knowledge base or some othersource. Thus, a query including the name of a person who is notwell-known may not be identified as an entity.

At step 720, an entity-details intent classification score is determinedfor the query using a machine classifier. The machine classificationsystem can be a neural network, support vector machine (SVP), or usesome other technology. Essentially, the score is a measure of howclosely the query matches training queries labeled as having anentity-details intent. The machine classification system is trainedusing queries labeled as having an entity-details intent or not havingan entity-details intent. Once trained, the machine classificationsystem can assign an intent classification score to an unlabeled query.The intent classification score can be a confidence score indicating adegree of confidence the machine classification system assigns to thequery that an entity-details experience should be shown in response tothe query. Conceptually, a high score means that the new query has ahigh level of similarity to training data labeled with an entity-detailsintent. A low score means the opposite.

At step 730, information about a question-and-answer experience thatwould be presented by the search engine in response to the query isidentified. The information can include a source from which thequestion-and-answer experience in generated, a classification for thesource, and other information. The information can also include acategory or type of question-and-answer experience generated. The typecan be form-based, for example, lists, or content category-based, suchas healthcare related.

At step 740, information about an entity-details experience that wouldbe presented by the search engine in response to the query isidentified. The information can include the title of the experience,content in the experience, and the like.

At step 750, an entity-details intent is determined to exist for thequery using a plurality of rules that take the entity-details intentclassification score, the information about the question-and-answerexperience, and the information about the entity-details experience asinput. The plurality of rules can include use of an intent scoregenerated by a classifier and other rules that do not use the intentscore.

In one aspect, two different intent-score based rules are used todetermine whether an entity-details intent exists. If either of the twointent-score based rules determines that entity-details intent exists,then an available Q&A experience is not presented and only theentity-details experience is presented. As described subsequently, otherrules that do not rely on the intent score may be used to determine thatentity-details intent exists.

The first intent-score based rule takes the intent score from theclassifier, usage information about the intent experience that could beshown in response to the query, and information about the Q&A experiencethat could be shown in response to the query as input. The informationabout the Q&A experience is a type classification. Q&A experiences canbe classified as many different types. Two types of particular interestare health information and lists. A health type of Q&A experienceprovides information related to a medical condition. For example,symptoms of scarlet fever may be presented in a Q&A experience inresponse to a query “scarlet fever.” A list of top grossing movies maybe presented as a Q&A experience in response to a query “top 10 moviesof 2018.” Other types of Q & A experiences include technicalinformation, financial information, date and time information, andweather. An exemplary date and time query could be, “when is Labor Day?”

In one aspect, the first intent-score-based rule requires threedifferent criteria to be met in order to find that an entity-detailsintent exists. First, the intent score from the classifier needs to beabove a high threshold score. This score is described as “high” inrelation to the lower threshold used in the second rule. Second, thesearch engine usage data must show that the entity experience has beenpresented previously in response to a similar query. Third, the Q&Aexperience type must not be on a white list. The white list includes Q&Aexperience types that should always be shown, which means that anentity-details intent should not be found. The white list can include atleast the health type, time/date, and list type of experiences. If allthree criteria are satisfied, then an entity-details intent is found.

The second intent-score-based rule uses the same input as the first,however, instead of the Q&A experience type, the rule uses the Q&Aexperience source. In one aspect, the second intent-score based rulerequires three different criteria to be met in order to find that anentity-details intent exists. First, the intent score from theclassifier needs to be above a medium threshold score. This score isdescribed as “medium” in relation to the higher threshold used in thefirst rule. The medium threshold may be set to be above a range thatdefines an ambiguous intent. For example, 0.6 (on a 0 to 1 scale) couldbe the medium threshold. In other words, the classifier shows anentity-details intent, but the signal does not need to be strong.Second, the search engine usage data must show that the entityexperience has been presented previously in response to a similar query.Third, the source of the Q&A experience must not be classified as acurated reference source, such as Wikipedia or IMDB. The technologydescribed herein can use a list of curated reference sources and use alookup function to determine if this third aspect of the rule issatisfied. In another aspect, if the entity-experience and the Q&Aexperience are from the same source, then the source of the Q&Aexperience is classified as a curated reference source. If all threecriteria are satisfied, then an entity-details intent is found.

In addition to the intent-score-based rules described previously,content-based rules may be used to determine that an entity-detailsintent exists. In one aspect, two different content-based rules are usedto determine whether an entity-details intent exists. If either of thetwo content-based rules determines that entity-details intent exists,then an available Q&A experience is not presented and only theentity-details experience is presented. Optionally, the intent scorecould be used with either content-based rule in the form of alow-threshold. The content-based rules form a strong presumption thatthe entity-details intent is present, thus, a lower threshold (than thepreviously described high and medium thresholds) could be used.

The first content-based rule determines whether a title of the entityexperience matches the query. If the title matches the query, then anentity-details intent is found. The match may be exact, n-gram based,and/or account for misspellings using a minimal edit distance.

The second content-based rule generates a similarity score thatquantifies the similarity between the content of entity experience andthe content of Q&A experience. If the similarity score is above athreshold, then an entity-details intent is found. The entity experienceoften includes more features, thus, the similarity score could be basedon an amount of content within the Q&A experience that is also in theentity experience. In other words, the similarity score may be generatedin a way that does not penalize the entity experience for havingadditional content.

Thus, aspects of the technology may use four different rules to find anentity-details intent. If any one or more of the rules is satisfied,then the intent is found. The rules may be evaluated in series or inparallel. If in series, then subsequent rules in the series need not beevaluated when a precedent rule finds an entity-details intent. If inparallel, then processing of other rules may stop when a rule finds anentity-details intent.

At step 760, an entity-details experience to the user is output. Theentity-details experience identifies the entity and multiple generalattributes that are not selected based on content in the query. Anexemplary entity-details experience has been described previously withreference to FIG. 4 .

With reference to FIG. 8 , computing device 800 includes a bus 810 thatdirectly or indirectly couples the following devices: memory 812, one ormore processors 814, one or more presentation components 816, one ormore input/output (I/O) ports 818, one or more I/O components 820, andan illustrative power supply 822. Bus 810 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 8 are shown with lines for the sakeof clarity, in reality, these blocks represent logical, not necessarilyactual, components. For example, one may consider a presentationcomponent such as a display device to be an I/O component. Also,processors have memory. The inventors hereof recognize that such is thenature of the art and reiterate that the diagram of FIG. 8 is merelyillustrative of an exemplary computing device that can be used inconnection with one or more aspects of the present technology.Distinction is not made between such categories as “workstation,”“server,” “laptop,” “handheld device,” etc., as all are contemplatedwithin the scope of FIG. 8 and with reference to “computing device.”

Computing device 800 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 800 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprisecomputer-storage media and communication media.

Computer-storage media includes both volatile and nonvolatile, removableand non-removable media implemented in any method or technology forstorage of information such as computer-readable instructions, datastructures, program modules, or other data. Computer storage mediaincludes, but is not limited to, RAM, ROM, EEPROM, flash memory or othermemory technology, CD-ROM, digital versatile disks (DVDs) or otheroptical disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bycomputing device 800. Computer storage media does not comprise signalsper se.

Communication media typically embodies computer-readable instructions,data structures, program modules, or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media, such as awired network or direct-wired connection, and wireless media, such asacoustic, RF, infrared, and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 812 includes computer storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 800includes one or more processors 814 that read data from various entitiessuch as memory 812 or I/O components 820. Presentation component(s) 816presents data indications to a user or other device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, and the like.

The I/O ports 818 allow computing device 800 to be logically coupled toother devices, including I/O components 820, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc.

The I/O components 820 may provide a natural user interface (NUI) thatprocesses air gestures, voice, or other physiological inputs generatedby a user. In some instances, inputs may be transmitted to anappropriate network element for further processing. An NUI may implementany combination of speech recognition, touch and stylus recognition,facial recognition, biometric recognition, gesture recognition both onscreen and adjacent to the screen, air gestures, head and eye tracking,and touch recognition associated with displays on the computing device800. The computing device 800 may be equipped with depth cameras, suchas stereoscopic camera systems, infrared camera systems, RGB camerasystems, and combinations of these, for gesture detection andrecognition. Additionally, the computing device 800 may be equipped withaccelerometers or gyroscopes that enable detection of motion. The outputof the accelerometers or gyroscopes may be provided to the display ofthe computing device 800 to render immersive augmented reality orvirtual reality.

Some aspects of computing device 800 may include one or more radio(s)824 (or similar wireless communication components). The radio 824transmits and receives radio or wireless communications. The computingdevice 800 may be a wireless terminal adapted to receive communicationsand media over various wireless networks. Computing device 800 maycommunicate via wireless protocols, such as code division multipleaccess (“CDMA”), global system for mobiles (“GSM”), or time divisionmultiple access (“TDMA”), as well as others, to communicate with otherdevices. The radio communications may be a short-range connection, along-range connection, or a combination of both a short-range and along-range wireless telecommunications connection. When we refer to“short” and “long” types of connections, we do not mean to refer to thespatial relation between two devices. Instead, we are generallyreferring to short range and long range as different categories, ortypes, of connections (i.e., a primary connection and a secondaryconnection). A short-range connection may include, by way of example andnot limitation, a Wi-Fi® connection to a device (e.g., mobile hotspot)that provides access to a wireless communications network, such as aWLAN connection using the 802.11 protocol; a Bluetooth connection toanother computing device is a second example of a short-rangeconnection, or a near-field communication connection. A long-rangeconnection may include a connection using, by way of example and notlimitation, one or more of CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

Many different arrangements of the various components depicted, as wellas components not shown, are possible without departing from the scopeof the claims below. Aspects of the present technology have beendescribed with the intent to be illustrative rather than restrictive.Alternative aspects will become apparent to readers of this disclosureafter and because of reading it. Alternative means of implementing theaforementioned can be completed without departing from the scope of theclaims below. Certain features and sub-combinations are of utility andmay be employed without reference to other features and sub-combinationsand are contemplated within the scope of the claims.

What is claimed is:
 1. A method for providing relevant search resultcontent in an experience format that is optimized to a query,comprising: receiving, at a search engine, a query from a user, thequery including an entity; determining an entity-details intentclassification score for the query using a machine classifier;identifying a type for a question-and-answer experience that would bepresented by the search engine in response to the query; determiningthat an entity-details intent is associated with the query when theentity-details intent classification score is above a threshold score;determining that the type for the question-and-answer experiencesatisfies a type criteria; and causing the question-and-answerexperience to be output to the user, rather than an entity-detailsexperience, because the question-and-answer experience satisfies thetype criteria.
 2. The method of claim 1, wherein the type criteria isthe question-and-answer experience matching a plurality ofquestion-and-answer types.
 3. The method of claim 2, wherein theplurality of question-and-answer types include a healthcare response anda list response.
 4. The method of claim 1, wherein the type criteria isthe question-and-answer experience being generated from a source on acurated source list.
 5. The method of claim 4, wherein the curatedsource list includes educational reference documents.
 6. The method ofclaim 1, wherein the method further comprises training the machineclassifier using labeled edge case training queries that are identifiedusing a preliminary classifier trained using a small set of labeledtraining queries, wherein the small set comprises less than 1/10 anumber of total training queries used to train the machine classifier.7. The method of claim 1, wherein the method further comprisesdetermining that the query matches a specific query within a hotfixlist, wherein the hotfix list includes one or more specific queries thatshould not be assigned an entity-experience intent.
 8. A method forproviding relevant search result content in an experience format that isoptimized to a query comprising: receiving, at a search engine, a queryfrom a user, the query including an entity; determining anentity-details intent classification score for the query using a machineclassifier; identifying a type for a question-and-answer experience thatwould be presented by the search engine in response to the query;determining that an entity-details intent is not associated with thequery when the entity-details intent classification score is below athreshold score; determining that the type for the question-and-answerexperience satisfies a type criteria; and causing thequestion-and-answer experience to be output to the user, rather than anentity-details experience, because the entity-details intent is notassociated with the query.
 9. The method of claim 8, wherein the typecriteria is the entity not exactly matching a title of theentity-details experience.
 10. The method of claim 8, further comprisingidentifying information about the question-and-answer experience thatwould be presented by the search engine in response to the query,wherein the type criteria is a similarity score between content in thequestion-and-answer experience having greater than a thresholdsimilarity to content in the entity-details experience.
 8. The method ofclaim 8, further comprising: identifying information about thequestion-and-answer experience that would be presented by the searchengine in response to the query, wherein the type criteria comprises thequestion-and-answer experience matching a plurality ofquestion-and-answer types on a list.
 9. The method of claim 8, furthercomprising: identifying information about the question-and-answerexperience that would be presented by the search engine in response tothe query, wherein the type criteria comprises the question-and-answerexperience being generated from a source on a curated source list. 10.The method of claim 8, wherein the machine classifier is a supportvector machine.
 11. The method of claim 12, wherein thequestion-and-answer experience includes a subset of content about theentity that is selected from a full set of content about the entitybased on terms in the query different from terms that identify theentity.
 12. The method of claim 8, wherein the method further comprisesdetermining that the query matches a specific query within a hotfixlist, wherein the hotfix list includes one or more specific queries thatshould not be associated with an entity-experience intent.
 13. One ormore computer storage media that, when executed by a computing device,causes the computing device to providing a relevant search resultcontent in an experience format that is optimized to a query, the methodcomprising: receiving, at a search engine, a query from a user, thequery including an entity; determining an entity-details intentclassification score for the query using a machine classifier;identifying information about a question-and-answer experience thatwould be presented by the search engine in response to the query;identifying information about an entity-details experience that would bepresented by the search engine in response to the query; determiningthat an entity-details intent is not associated with the query using aplurality of rules that take the entity-details intent classificationscore, the information about the question-and-answer experience, and theinformation about the entity-details experience as input; and outputtingthe question-and-answer experience to the user.
 14. The media of claim13, wherein one of the plurality of rules comprises the entity-detailsintent classification score being above a threshold score and thequestion-and-answer experience matching a plurality ofquestion-and-answer types on a curated source list.
 15. The media ofclaim 16, wherein one of the plurality of rules comprises theentity-details intent classification score being below a thresholdscore.
 16. The media of claim 13, wherein one of the plurality of rulescomprises the entity not exactly matching a title of the entity-detailsexperience.
 17. The media of claim 16, wherein one of the plurality ofrules comprises a similarity score between content in thequestion-and-answer experience not having greater than a thresholdsimilarity to content in the entity-details experience.